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ABSTRACT 


The Basket Method and Dollar Unit Sampling techniques are examined to 
determine which of them draws a sample which allows a more accurate estimate of a 
population to be made. The several populations used in the simulation have errors 
planted to represent both random “honest” mistakes and weighted “dishonest” 
mistakes. The author concludes that the Basket Method has a more desireable 


accuracy pattern than the Dollar Unit Sampling Technique. 
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I. INTRODUCTION 


A. NEED FOR THIS STUDY 

In major weapon system acquisition, the Department of Defense typically issues 
a large number of change orders to modify an existing contract. In each case, the 
contractor prepares a proposal of contract price adjustment for the requested changes. 
A lead ship or aircraft production contract may generate over 10,000 such proposals. 

Acquisition regulations require the government to analyze each proposal and 
negotiate with the contractor for a fair price adjustment. Hlowever, these requirements 
produce more work than the existing number of government auditors can undertake. 
Generally, as backlogs develop, thoroughness is reduced for expediency, and errors slip 
by. However, work in progress usually goes on, but contractors are not paid until after 
the proposal is processed. This delay in funding raises the capital cost to the 
contractor for performing work for the government. 

It is in the best interests of both the government and the contractors to raise the 
accuracy and speed and reduce the cost of auditing such proposals of price adjustment. 
If it were allowed by the acquisition process, analyzing and negotiating a sample of 
proposals selected with an effective sampling technique could have these effects. 

The reason for auditing the proposal population is to ensure the proposals are 
realistic and fair. During an audit, a proposal will be found to be understated, that 1s 
the Government feels that the proposed cost of a change to the contract is less than 
the actual cost the contractor will incur, or overstated, or correct. Any overstatement 
Or understatement is considered an error in the proposal population. The ideal 
sampling technique would choose a sample from the population of proposals which, 
when audited, would always give an estimated value for the population as a whole 
which was exactly correct, no matter what the degree or distribution of errors in the 
proposal population. Such a technique does not exist. 

Assuming, then, that there will be some degree of error in the prediction of the 
true value of the entire proposal population whenever a sampling technique is used; the 
behavior of the degree of error must be predictable and exhibit certain qualities in 
order for the sampling technique to be considered appropriate for the purpose 
described above. 


Specifically, the degree of error should not be easily altered by the error 
arrangement, that is the distribution, size, frequency, or type (overstatement or 
understatement) of errors found in the population, as long as the total error ts the 
same. A sampling procedure which shows little response to changes in the error 
arrangement will be said to show consistency. If certain patterns of errors caused the 
entire population to be evaluated as understated then the government would pay more 
than a fair price for the changes described by the population of proposals which 
contained those errors. This, in turn, would tend to encourage fraud, since a 
contractor could carefully seed his proposal population with deliberate errors of the 
appropriate size, type, and distribution and thereby be awarded a larger payment from 
the government. 

Therefore, the desired method will not necessarily be the one which results in the 
most accurate, average estimate for various error arrangements in the proposal 
population. It will instead be the method which responds least to variations in the 


error arrangement. 


B. METHODOLOGY 

In this paper the Dollar Unit Sampling Method and the Basket Method will be 
used to draw samples from populations for evaluation. The populations were used 
earlier in a joint study of the American Institute of Certified Public Accountants and 
the American Statistical Association. Each population consists of two columns of 
values which represent the proposed, or book value of a contract change and the 
audited or true value of that change. The populations are rigged with either random or 
planned errors. Samples drawn by the two methods from each population will be 
evaluated and compared to determine which method gives a better estimate of the 
whole population according to the goals described above. Both the error rigging and 
evaluation steps are explained further in the description of the simulation. 

The amount of work associated with auditing is more closely related to the 
number of items being audited than to the total dollar value of all the items being 
audited. Therefore, the sampling rules of the two methods will be adjusted so that they 
will draw samples with the same number of proposals from each population. The 
results of this study will then indicate which method yields the more desirable 


prediction, that is, the greater benefit, while holding the cost of the audit constant. 


Il. DOLLAR UNIT SAMPLING 


A. HISTORY 

Dollar Unit Sampling is a technique which is strongly based on statistical theory. 
It determines the possible error in a sample based on a Poisson model of the 
population. 

The roots of the Dollar Unit Sampling method can be traced to the 1960’s when 
van Heerden developed the “guilder number” method [Ref. 1]. About the same time, 
K. W. Stringer developed a similar model which he called Cumulative Monetary 
Amounts (CMA) Sampling [Ref. 2]. Dr. Albert Teitlebaum then developed a model 
similar to Cumulative Monetary Amounts Sampling which he called Dollar Unit 
Sampling (DUS) [Refs. 3,4,5,6]._ In December of 1973 he presented the first attempt at 
a comprehensive proof for Dollar Unit Sampling, which led to rising prominence of the 
Dollar Unit Sampling method [Refs. 3,7]. 

Dollar Unit Sampling is also known as Monetary Sampling (MS), Combined 
Attribute Variable Sampling (CAV), Monetary Unit Sampling Technique (MUST), 
Sampling Proportionate to size (SPS), and Probability Proportional to Size (PPS) 
[Ref. 8]. 

The CMA procedure is the most widely used version of DUS, but it has the 
disadvantage of overestimating the possible overstatement [Refs. 7,8,9]. While this is a 
conservative error, from the government’s viewpoint, it is still an error, and contractors 
will be unlikly to bind themselves to a method which will routinely overcorrect against 


them. 


B. DESCRIPTION OF DOLLAR UNIT SAMPLING 

There are three steps to using the Dollar Unit Sampling method: calculating the 
sample size, choosing the sample, and calculating the error. These steps will be briefly 
discussed below. 

1. Calculating Sample Size 

There are four commonly used methods for the determination of the sample 

size. All require the total book value of the contract changes, the reliability factor for 
a given level of confidence, and the amount of error which can be tolerated for a given 


level of confidence. The better the estimated amount of error in the sample is known, 


the more accurate the required sample size will be. In practice, tables are used which 
specify the sample size based certain arguments for book value, level of confidence, and 
estimated error rate. 

In this study; however, the process of selecting a sample size is not required to 
be followed. This is because in actual applications where the true value of the 
population is not known the only way to be reasonably certain that one’s results are 
valid is by complying with rules which will tie the audit to statistical theory. The 
sampling rules for Dollar Unit Sampling are intended to do just that, so that the 
auditor who follows the sampling procedures will be able to determine the extent of the 
audit required to achieve the desired certainty. 

In this study the true values of the proposals are known, as are the size and 
distribution of the errors, and the Dollar Unit Sampling method is not being compared 
to its theoretical limits, but to a second method to determine which of the two gives a 
more desireable result in a certain case for the same amount of work. Therefore, the 
size of the sample must be based on a number of proposals rather than a number of 
dollars. This being the case, one tenth of the total number of proposals in the 
population was arbitrarily chosen as the size. 

2. Drawing the Sample 

The heart of Dollar Unit Sampling is the choosing of the sample. Rather than 
a random technique, which might select a certain number of contracts to audit, the 
sample size calculated in the previous step refers to a number of single dollar units. 
Single dollar units are merely dollars. Thus, a proposal which is estimated at one 
hundred dollars would contain one hundred single dollar units. Each of these dollars 
has the same probability of being selected as every other dollar in the entire proposal 
population. When a dollar unit is selected, the entire proposal in which it resides 1s 
audited. Since the single dollar units are each given the same chance of being selected, 
a proposal with a larger dollar value than another has a larger chance of being selected 
for the audit. 

To select the dollars, and thereby the proposals, to be audited, the proposals 
are stacked and then added to form a series of partial sums. The proposals which lie 
between the partial sums which bracket the random numbers generated by the sizing 
calculations are selected for auditing. A simple algorithm for accomplishing this 1s to 
subtract the proposal book values from the random number until the difference 1s 
negative and audit the last proposal subtracted. The step by step process is as follows: 


1. Arrange the proposals in a sequence and find the sum of their book values. 


e 


2. Calculate the Average Sample Interval (ASI) using the equation: 
ASI = B/n 

where 

B = sum of book values and . 

n = number of dollars to be chosen as determined by the sample size calculation 
Select a random start number RS such that 0 < RS < ASI. 


4. Subtract the proposal book, values, in the sequence determined in step one, 
from RS until the difference 1s negative. 


5. Audit the proposal which makes the running difference negative, add the ASI to 
the negative difference, and repeat from step 4. 


If a proposal has a book value greater than the ASI it will be chosen. A 
proposal is only allowed to be chosen once, no matter how large it is, so the number of 
dollar units calculated when determining the sample size will be greater than number of 
proposals selected for audit if any proposals have a book value greater than the ASI. 
The development of digital computers has greatly eased the execution of this phase. 

3. The Peach Problem 

Before discussing the evaluation phase, it is useful to briefly illustrate and 
develop the Dollar Unit Sampling technique by examining the ‘Peach Problem’ [Ref. 8]. 
In the Peach Problem one must determine the portion of peaches in a lot of one 
million which is unusable. To determine the size of the portion one will draw a sample 
and use information on both the number of rotten peaches and the extent of rottenness 
found. 

One hundred sample peaches are randomly drawn and examined. If there are 
no defective peaches, there is a 5% risk that the percentage of peaches defective 
exceeds 3%. These numbers are based on the Poisson distribution which is chosen 
because it is a good estimater of the binomial case being examined. The binomial is 
that a peach 1s either rotten to some detectable degree or it is not. Similarly, a 
proposal is either overstated or understated to some significant degree or it is accurate. 
It should also be noted that the terms mentioned in the sample size calculation now 
take on values. The total book value is one million. The @ risk, or risk of accepting 
an unacceptable lot, is 5%, which means the confidence 1s 95%. The reliability factor 
for the 95% confidence level is 3%, meaning that at most, 30,000 peaches could be 
rotten to some degree and the population would still be acceptable. 

If there had been one defective peach, there would have been a 5% @ risk that 


the percentage of defective peaches exceeded 4.75%. Similarly, if two defective peaches 
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were found, there would have been a 5% @ risk that the number of defective peaches 
will exceed 6.31%. These values also come from the Poisson distribution. 
What is not considered yet is the fraction of the defective sample peaches 
which is rotten. This is important if one can use a partial peach or partial dollar. 
To account for this in the case with two defective peaches in the sample, the 
two defective peaches are examined. Assume the two rotten peaches were 8% and 4% 
rotten. Then in 95% of the cases: 
1. The number of totally rotten peaches will be less than 3%. 
2s Mg number of 8% rotten peaches will not exceed an additional 1.75%. (4.75% 


a 0 
3. The number of 4% rotten peaches will not exceed 1.56%. (6.31% - 4.75%) 


Then the total portion of the population which is rotten will be less than: 


3.00% x 100% =. 3.00% 
175% x 8% = 0.14% 
1.56% x 4% = 0.06% 

4.62% 


This is a better estimate of the total amount of rotten peaches in the lot than 
6.31% because it corrects for the usable part of the partially rotten or ‘tainted’ 
peaches. It should be noticed that the additional corrections are arranged in order of 
decreasing percent tainting. This gives the most conservative result since the smaller 
percent taintings are applied to the smaller percent defect increments. 

The analogy applies directly to the proposals problem. The peaches are 
dollars, each one essentially the same size. The degree of rottenness is the degree of 
error in the entire proposal in which the sampled dollar is found. Thus, if a chosen 
dollar is in a proposal which, when audited, reveals an 8% overstatement error, each 
dollar in that proposal, and in particular, the dollar which was chosen to be sampled, is 
considered 8% wasted. 

4. Calculation of the Result 

If there are no errors in the sample, no calculation is required and the amount 
of error for the planned level of confidence is the planned, tolerable amount of error. 

Similarly, if the sample size was determined using the ff Risk Only tables, and 
the sum of the taintings in the sample is less than that planned when the sample size 
was determined, then the resulting tolerable amount of error is less than that planned 


and no calculation is required. 


If the sample size was determined using the @ and f risk tables, and the sum 
of the taintings is less than the K-factor,! then the actual error is less than the planned, 
tolerable error limit; three percent in the previous example. 

If some other result is reached then some calculation will be necessary. Each 
erroneous proposal is listed in order of decreasing tainting. The taintings are then 
multiplied by the incremental percent defect. The sum of the taintings and tainting 
products are then used to compute the met upper error limits. 

If any errors are found materiality must be redefined since the sample size 
anticipates that the sample will contain no errors. If the errors found are few or small, 
the new materiality might be close enough to the original goal to allow acceptance of 


the population. 


[The K-factor is one of the two values read from the tables used to determine the 
sample size, the other being the sample size. The K-factor is significant because it is 
compared to the sum of the tainting in the sample to determine if the population can 
be accepted without exceeding the B risk y taintings S K-factor) or rejected without 
EXCecdime means ke > taintings > K-factor) 


IZ 


Il. THE BASKET METHOD 


A. HISTORY 

The Basket Method was developed by Dr. K. T. Wallenius under the sponsorship 
of the Office of Naval Research and the Naval Material Command. It is a recently 
created technique which was formed in direct response to the problems of auditing and 


negotiating numerous contract proposals with sole source contractors [Ref. 10]. 


B. DESCRIPTION 

The Basket Method derives its name from its process of sorting a population into 
a number of baskets, one of which will be randomly selected for sampling. The aim of 
the basket assignment process is to make each basket resemble the population in any 
traits of interest. In general, the spread and proportion of proposal values will be 
nearly identical to the population. Sophisticated software can quickly balance baskets 
for work center assignments, material usage, skill requirements, etc. In short, whatever 
trait is deemed pertinent to the value of an audit will be balanced by the Basket 
Method where possible. Obviously, in a population of proposals where one is ten times 
larger than any others it would be more practical to remove that proposal from the 
population before assigning the baskets. 

1. Basket Assignment 

Assignment of a population to baskets follows a simple set of rules: 


i os the proposals in the population are arranged in descending order by book 
value. 


2. The largest unassigned proposal is assigned to the first basket, the second 
largest to the second basket, and so on for the first proposals. 


os) 


The baskets are sorted in ascending order by total assigned proposal book value. 
ee oe time this step is performed it will simply reverse the order of the 
askets. 


4. The eh nment and sort steps are repeated until all the proposals are assigned 
to a basket. 


The effect of this procedure is to add the largest of the next n unassigned 
proposals to the basket with the smallest total book proposal value. The end result 1s 
that each basket will have nearly identical total book values and proposal size mixes. 
The software previously noted will make adjustments to correct for any deviations in 


totals and balance the mix for traits other than cost as well. 
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2. Evaluation of Baskets 
One basket is chosen randomly and all its proposals are audited. The 
resulting audit value of the basket 1s divided by the proposal value to produce a 
correction factor. The proposal value of the population is then multiplied by the 
correction factor to determine the population audit result. This will be the estimated 


true value of the population. 
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IV. DESCRIPTION OF SIMULATION 


A. DERIVING COMPARABLE RESULTS 

Direct comparison of the Basket Method and the Dollar Unit Sampling method 
will not be meaningful since the former yields a best estimate of the error rate of a 
population and the latter returns an upper error limit for a stated assurance level. 
Since it is the consistency of the drawn sample which 1s of interest in this investigation, 
the sample selection process of both the Basket Method and the Dollar Unit Sampling 
Technique will be used to draw samples from the same populations. The samples will 
then be evaluated according to the Basket Method, which will give an estimate of the 
true value of the population, to see how well the sample produced by each method 
reflects the value of the population. This is appropriate since the negotiated price of a 
contract should be a best estimate, not an upper limit. 

The rules of the Basket Method will create the identical set of baskets from a 
given population every time they are applied. Therefore, the baskets of the Basket 
Method can easily be evaluated by complete enumeration. The rules of the Dollar 
Unit Sampling technique also provide a finite number of samples, but that number 1s 
much larger than the number of different baskets. Therefore, the Dollar Unit Sampling 
technique will be evaluated using a Monte Carlo process in which a random start 
number is generated for each run and the ASI and order of proposals are kept 
constant. 

To evaluate the sample drawn with the Dollar Umit Sampling rules using the 
Basket Method evaluation technique the sample is treated as if it was a basket. All its 
resident proposals are audited and their true value is divided by their proposal value. 
The resulting factor is multiplied against the population proposal total to determine the 


best estimate of the true total value of the proposal population. 


B. PREPARATION OF THE TEST POPULATIONS 

The populations used are described in Table I. 

The original populations were randomly seeded with errors at a 5% and 10% 
rate of occurance. The 5% population was then skewed to form two additional test 
populations. One had its errors skewed strongly to its higher valued proposals, the 


other to its lower valued proposals. The total dollar amount of error and number of 


Ie 


PAB a 


POPULATION DESCRITIIO 


NAME A B C D e! 
Population 8,300 8,300 8,300 8,300 8,300 
Erroneous Proposals 5% 5% 5% 10% 10% 
¥ $ Errors/), S$ Proposals 9% 9% 9% 0.1% 10% 
Types of Errors 15 ae 2 2 + 
Skew (none, high, or low) N H Er N N 
NAME AA BB ae DD BE 
Population 8,300 8,300 8,300 8,300 8,300 
Erroneous Proposals a6 5% 5% 10% 10% 
y* § Errors/}” $ Proposals 9% 9% 9% 0.1% 10% 
Types of Errors + |- + /- + /- aie + /- 
Skew (none, high, or low) N H L N N 


overstated proposals remained constant during the skewing process. All populations 
were created in both a “dishonest” version which had overstatements only and are 
named by single letters, and in an “honest” version with both overstatements and 
understatements named by double letters. Except for the sign on each error, the single 
letter named populations are identical to their double letter named counterparts. The 
populations with 10% errors differ in that E and EE, while containing the same 
number of errors in the same distribution and sign as D and DD respectively, have 
errors of much larger magnitude, so that the sum of the dollar value of the errors make 


up 10% of the population in E and EE but only 0.1% of the population in D and DD. 


C. SIMULATION EXECUTION 

Simulations were run on all populations using both a Basket Method evaluating 
program and a Dollar Unit Sampling monte carlo simulation program. First, ten 
baskets were chosen for the Basket Method to remain consistent with the earlier 
arbitrary decision to audit one tenth of the proposals. This resulted in 830 proposals 
per basket. Next, an ASI was then selected to produce an average of 830 proposals in 
the Dollar Unit Sampling simulation, and ten trials were run using the Dollar Unit 
Sampling selection method. The programs, written for this study in Waterloo Basic, 


are listed in Appendix A. 
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The ten audit results for each method were then divided by their respective 
proposal sums to derive the correction factors. These correction factors were 
multiplied by the sum of all proposals to give the predicted true total for the 
population. The difference between the predicted and the actual total was then divided 
oy the sum of all proposals to give a percent error for each basket and trial. The 


means of the percent errors for each method and each population are listed in Table 2. 


Jie LE 2 
SIMULATION RESULTS (MEAN % ERROR) 


NAME A B C D E 

Basket Method 0.77 1.03 0.66 0.14 035 
DUs 2.40 -2.06 peas: 0525 4.75 

NAME AA BB Ce DD Be 

Basket Method 0299 7 1.16 0.15 et 
DUS Ons 0.14 -0.01 -0.08 -0.28 


Detailed results are shown in Appendix B. A positive percent error represents an 
overestimate and a negative number represents an underestimate error. Using the 
Basket Method to draw a sample was always more accurate against the overstatements 
only padding technique. Using the Dollar Unit Sampling technique to draw the sample 


was more accurate when the proposals contained errors in both directions. 


D. RESISTANCE TO PROPOSAL RIGGING 

In order to benefit from the potential time and labor savings a sampling system 
offers, the sampling technique must be resistant to padding schemes. If not, a 
contractor has much to gain by manipulating the errors in his proposals. Therefore, as 
mentioned previously, the central issue 1s not necessarily to determine which of these 
two methods is the most accurate, but to discover which one benefits padding schemes 
least. When comparing a method’s performance between the single and double letter 
versions of a population it can be seen that the Basket Method 1s essentially as strict or 
stricter, from the governments view, when estimating the value of the overstatement 
only population as it is when estimating the value of the honest population. For 
example, the A population, which was rigged with overstatements only, was evaluated 


by the Basket Method at a value exceeding its actual value by 0.77%. The AA 


population, with randomly decided over and understatements, was evaluated at a value 
exceeding its true value by 0.99%. If these two populations had been submitted, the 
contractor who made overstatements only would have received 0.22% less in payment 
from the government. In other words, padding one’s proposed contract changes with 
overstatements in random, low, or high skewed distributions prior to submitting them 
to the Basket Method for evaluation is not likely to raise the resulting estimate for the 
population but is instead likely to lower the estimated value. On the other hand, the 
sample drawn with Dollar Unit Sampling allowed the overstatement padded 


populations a significantly larger estimate than it did the honest populations. 


E. EVALUATION 

Assuming honest contractors are as likely to understate as overstate their costs 
and dishonest contractors are not, honest contractors will be more successful than 
dishonest contractors against the Basket Method, and dishonest contractors will be 
more successful than honest contractors against the Dollar Unit Sampling sample 
method. To choose the Dollar Unit Sampling method to determine a contract value 
would, in effect, encourage overbidding, since it would be more profitable than 


remaining honest. Under the Basket Method honesty is as profitable as dishonesty. 


F. AREAS FOR FURTHER RESEARCH 

This study may be expanded in several ways. It may be repeated with fewer or 
more baskets. A data set with a much smaller variance in proposal size may be used. 
The Basket Method can be compared to additional sampling methods. Additional 
Strategies for error arrangements which will reduce the accuracy of the Basket Method 
can be developed and tested. Particularly, will an understatement only strategy with 
the same magnitude and distribution of proposal errors result in a total population 


error in equal magnitude and opposite sign as an overstatement strategy. 
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APPENDIX A 
SIMULATION PROGRAM LISTINGS 


Basket Method Enumeration Program Listing 
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Dollar Unit Sampling Monte Carlo Program Listing 
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APPENDIX B 
DETAILED RESULTS 


POPULATION A 
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